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Abstract 

English words and the outputs of many other natural processes are well-known to follow 
a Zipf distribution. Yet this thoroughly-established property has never been shown to help 
compress or predict these important processes. We show that the expected redundancy of 
Zipf distributions of order a > 1 is roughly the l/a power of the expected redundancy of 
unrestricted distributions. Hence for these orders, Zipf distributions can be better compressed 
and predicted than was previously known. Unlike the expected case, we show that worst- 
case redundancy is roughly the same for Zipf and for unrestricted distributions. Hence Zipf 
distributions have signihcantly different worst-case and expected redundancies, making them 
the first natural distribution class shown to have such a difference. 

Keywords: Power-law, Zipf, Universal Compression, Distinct elements. Redun¬ 
dancy 


1 Introduction 

1.1 Definitions 

The fundamental data-compression theorem states that every discrete distribution p can be com¬ 
pressed to its entropy H{p) ^ compression rate approachable by assigning each 

symbol x a codeword of roughly log bits. 

In reality, the underlying distribution is seldom known. For example, in text compression, we 
observe only the words, no one tells us their probabilities. In all these cases, it is not clear how to 
compress the distributions to their entropy. 

The common approach to these cases is universal compression. It assumes that while the un¬ 
derlying distribution is unknown, it belongs to a known class of possible distributions, for example, 
i.i.d. or Markov distributions. Its goal is to derive an encoding that works well for all distributions 
in the class. 

To move towards formalizing this notion, observe that every compression scheme for a distribu¬ 
tion over a discrete set A corresponds to some distribution q over A where each symbol a; £ A is 
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assigned a codeword of length log Hence the expected number of bits used to encode the dis¬ 
tribution’s output is ^p{x) log and the additional number of bits over the entropy minimum 
is Y.P{x)\og^. 

Let P be a collection of distributions over X. The collection’s expected redundancy, is the least 
worst-case increase in the expected number of bits over the entropy, where the worst case is taken 
over all distributions in V and the least is minimized over all possible encoders, 

RiV) min max p{x) log 


An even stricter measure of the increased encoding length due to not knowing the distribu¬ 
tion is the collection’s worst-case redundancy that considers the worst increase not just over all 
distributions, but also over all possible outcomes x, 


R{V) 


def 


min max max log 
9 pGV x£X 


p{x) 

q{x)' 


Clearly, 

R{V) < R{V). 

Interestingly, until now, except for some made-up examples, all analyzed collections had extremely 
close expected and worst-case redundancies. One of our contributions is to demonstrate a practical 
collection where these redundancies vastly differ, hence achieving different optimization goals may 
require different encoding schemes. 

By far the most widely studied are the collections of i.i.d. distributions. For every distribution 
p, the i.i.d. distribution p” assigns to a length-n string x"' (xi,X 2 ,... ,Xn) probability p(x"’) = 

p{xi) ■ ... ■ p{xn). For any collection V of distributions, the length-n i.i.d. collection is 




1.2 Previous results 

Let Afe denote the collection of all distribution over {1,... ,k'\, where A was chosen to represent 
the simplex. For the first few decades of universal compression, researchers studied the redundancy 
of A^ when the alphabet size k is fixed and the block length n tends to inhnity. A sequence 
of papers Krichevsky and Trofimov [1981], Kieffer [1978], Davisson [1973], Davisson et al. [1981], 
Willems et al. [1995], Xie and Barron [2000], Szpankowski and Weinberger [2010], Orlitsky and Santhanam 
[2004], Rissanen [1996], Cover [1991], Szpankowski [1998], Szpankowski and Weinberger [2012] showed 
that 

^ log ^ + log + Ofc(l), 

and that the expected redundancy is extremely close, at most log e bits lower. Note that a similar 
result holds for the complementary regime where n is fixed and k tends to infinity, 

k 

R(^k) = nXog —h o(n). 
n 
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These positive results show that redundancy grows only logarithmically with the sequence length 
n, therefore for long sequences, the per-symbol redundancy diminishes to zero and the underlying 
distribution needs not to be known to approach entropy. As is also well known, expected redundancy 
is exactly the same as the log loss of sequential prediction, hence these results also show that 
prediction can be performed with very small log loss. 

However, as intuition suggests, and these equations confirm, redundancy increases sharply with 
the alphabet size k. In many, and possibly most, important real-life applications, the alphabet size 
is very large, often even larger than the block length. This is the case for example in applications 
involving natural language processing, population estimation, and genetics Chen and Goodman 
[1996]. The redundancy in these cases is therefore very large, and can be even unbounded for any 
sequence length n. 

Over the last decade, researchers therefore considered methods that could cope with compression 
and prediction of distributions over large alphabets. Two main approaches were taken. 

Orlitsky et al. [2004] separated compression (and similarly prediction) of large-alphabet se¬ 
quences into compression of their pattern that indicates the order at which symbols appeared, and 
dictionary that maps the order to the symbols. For example, the pattern of ^^banana” is 123232 and 
its dictionary is 1 —)■ 6, 2 —)• a, and 3 —)• n. Letting denote the collection of all pattern distribu¬ 
tions, induced on sequences of length n by all i.i.d. distributions over any alphabet, a sequence of 
papers Orlitsky et al. [2004], Shamir [2006, 2004], Garivier [2009], Orlitsky and Santhanam [2004], 
Acharya et al. [2012, 2013] showed that although patterns carry essentially all the entropy, they 
can be compressed with redundancy 

0.3 • < ii(A;) < R{A^) < • log^ n 

as n —)• oo. Namely, pattern redundancy too is sublinear in the block length and most significantly, 
is uniformly upper bounded regardless of the alphabet size (which can be even inhnite). It follows 
the per-symbol pattern redundancy and prediction loss both diminish to zero at a uniformly- 
bounded rate, regardless of the alphabet size. Note also, that for pattern redundancy, worst-case 
and expected redundancy are quite close. 

However, while for many prediction applications predicting the pattern suffices, for compression 
one typically needs to know the dictionary as well. These results show that essentially all the 
redundancy lies in the dictionary compression. 

The second approach restricted the class of distributions compressed. A series of works stud¬ 
ied class of monotone distributions Shamir [2013], Acharya et al. [2014a]. Recently, Acharya et al. 
[2014a] showed that the class of monotone distributions over {1,... ,k} has redundancy R{M^) < 
^/20nlogklogn. 

More closely related to this paper are envelope classes. An envelope is a function / : N+ —)• M>o. 
For envelope function /, 

{p ■ Pi < f{i) for all i > 1} 

is the collection of distributions where each pi is at most the corresponding envelope bound /(i). 
Some canonical examples are the power-law envelopes f{i) = c-i~^, and the exponential envelopes 
f{i) = c- In particular, for power-law envelopes Boucheron et al. [2009, 2014] showed 

R{£f) < (logn)^"- +0(1), 
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and more recently, Acharya et al. [2014b] showed that 

R{£f) = 0(nV“). 

The restricted-distribution approach has the advantage that it considers the complete sequence 
redundancy, not just the pattern. Yet it has the shortcoming that it may not capture relevant 
distribution collections. For example, most real distributions are not monotone, words starting 
with ‘a’ are not necessarily more likely than those starting with ‘b’. Similarly for say power-law 
envelopes, why should words in the early alphabet have higher upper bound than subsequent ones? 
Thus, words do not carry frequency order inherently. 

1.3 Distribution model 

In this paper we combine the advantages and avoid the shortfalls of both approaches to compress 
and predict distributions over large alphabets. As in patterns, we consider useful distribution 
collections, and like restricted-distributions, we address the full redundancy. 

Envelope distributions are very appealing as they effectively represent our belief about the 
distribution. However their main drawback is that they assume that the correspondence between 
the probabilities and symbols is known, namely that pi < f(i) for the same i. We relax this 
requirement and assume only that an upper envelope on the sorted distribution, not the individual 
elements, is known. Such assumptions on the sorted distributions are believed to hold for a wide 
range of common distributions. 

In 1935, linguist George Kingsley Zipf observed that when English words are sorted according 
to their probabilities, namely so that > P 2 Y ..., the resulting distribution follows a power law. 
Pi ^ ^ for some constant c and power a. Long before Zipf, Pareto [1896] studied distributions 
in income ranking and showed it can be mathematically expressed as power-law. Since then, 
researchers have found a very large number of distributions such as word frequency, population 
ranks of cities, corporation sizes, and website users that when sorted follow this Zipf-, or power-law 
Zipf [1932, 1949], Adamic and Huberman [2002]. In fact, a Google Scholar search for “power-law 
distribution” returns around 50,000 citations. 

A natural question therefore is whether the established and commonly trusted empirical obser¬ 
vation that real distributions obey Zipf’s law can be used to better predict or equivalently compress 
them, and if so, by how much. 

In Section 2 we state our notation followed by new results in Section 3. Next, in Section 4 
we bound the worst-case redundancy for power-law envelop class. In Section 5 we take a novel 
approach to analyze the expected redundancy. We introduce a new class of distributions which 
has the property that all permutations of a distribution are present in the class. Then we upper 
and lower bound the expected redundancy of this class based on the expected number of distinct 
elements. Finally, in Section 6 we show that the redundancy of power-law envelop class can be 
studied in this framework. 

2 Preliminaries 

2.1 Notation 

Let x” {xi,X 2 ,.., Xn) denote a sequence of length n, X be the underlying alphabet and k \X\. 
The multiplicity of a symbol x G A is the number of times x appears in x"'. Let [k] = 
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{1, 2, k] be the indices of elements in X. The type vector of x” over [k] = {1, 2,A;}, r(x") = 
• • • ik-k) is a fc-tuple of multiplicities in x"". The prevalence of a multiplicity fi, denoted by 
is the number of elements appearing p, times in x^. For example, (pi denotes the number of 
elements which appeared once in x”. Furthermore, denotes the number of distinct elements in 
x”. The vector of prevalences for all //’s is called the profile vector. 

We use to note the highest probability in p. Hence, p(i) > p{ 2 ) > • • ■P(k)- Moreover, we 
use zipf(a, k) to denote Zipf distribution with parameter a and support k. Hence, 

zipf(a, A:)i = ——, 

^k,a 

where Ck^a is the normalization factor. Note that all logarithms in this paper are in base 2 and we 
consider only the case a > 1. 

2.2 Problem statement 

For an envelope / with support size k, let T(j-) be the class of distributions such that 

^(/) = {P ■ P{i) < fii) VI < i < k}. 

Note that Sf C £{f)- We also consider the special case when / is a distribution itself, in which 
case we denote by , a class that has distributions whose multi-set of probabilities is same 
as p. In other words, V(j,) contains all permutations of distribution p. Also we define 

V2 = {p^ : < d}, 

where is the number of distinct elements in x”. Note that for any distribution belonging to this 
class, all permutations of it are also in the class. 

3 Results 

We first consider worst-case redundancy, lower-bound it for general unordered permutations, and 
apply the result to unordered power-law classes, showing that for n < 

This shows that the worst-case redundancy of power-law distributions behaves roughly as that of 
general distributions over the same alphabet. 

More interestingly, we establish a general method for upper- and lower-bounding the expected 
redundancy of unordered envelope distributions in terms of expected number of distinct symbols. 
Precisely, for a class V2 we show the following upper bound 

Ri'Pd) <dlog^ + (21oge l)d + log(n 1). 

Interpretation: This upper bound can be also written as 

log n -I- log -I- log 


n — 1 
d-1 
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( 1 ) 




This suggests a very clear intuition of the upper bound. We can give a compression scheme for 
any sequence that we observe. Upon observing a sequence x”, first we declare how many distinct 
elements are in that sequence. For this we need log n bits. In addition to those bits, we need log (^) 
bits to specify which d distinct elements out of k elements appeared in the sequence. Finally, for 
the exact number of occurrences of each distinct element we should use log (^Zi) bits. 

We also show a lower bound which is dependent on both the expected number of distinct 
elements d and the distributions in the class V^- Namely, we show 

RiV^) > Hog n] - dlog ^ - dlogyre j (1 + 0^(1)) - ^ {Snpi - npilognpi). 

^ ^ d ' npi<0.7 

Using this result, we then consider expected redundancy of power-law distributions as a special 
case of V2 and show that it is significantly lower than that of general distributions. This shows that 
on average, Zipf distributions can be compressed much better than general ones. Since expected 
redundancy is the same as log loss, they can also be predicted more effectively. In fact we show 
that for k > n, 

Ri£{ci-‘=‘,k)) = 0(n-log/c). 

Recall that general length-n i. i. d. distributions over alphabet of size k have redundancy roughly 
n log ^ bits. Hence, when k is not much larger than n, the expected redundancy of Zipf distributions 
of order a > 1 is the 1/a power of the expected redundancy of general distributions. For example, 
for a = 2 and k = n, the redundancy of Zipf distributions is Q{^/nlogn) compared to n for general 
distributions. This reduction from linear to sub-linear dependence on n also implies that unordered 
power-law envelopes are universally compressible when k = n. 

These results also show that worst-case redundancy is roughly the same for Zipf and general 
distributions. Comparing the results for worst-case and expected redundancy of Zipf distributions, 
it also follows that for those distributions expected- and worst-case redundancy differ greatly. This 
is the first natural class of distribution for which worst-case and expected redundancy have been 
shown to significantly diverge. 

As stated in the introduction, for the power-law envelope /, Acharya et al. [2014b] showed that 

R{£f) = 0(n^/"). 

Comparing this with the results in this paper reveals that if we know the envelop on the class 
of distributions but we do not know the true order of that, we have an extra multiplicative factor 
of log k in the expected redundancy, i.e. 

R{£{f)) = 0(n^/" log A:). 


4 Worst-case redundancy 

4.1 Shtarkov Sum 

It is well known that the worst-case redundancy can be calculated using Shtarkov sum Shtarkov 
[1987], i.e. for any class V 

R{V)=logS{V), (2) 
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where S((P) is the Shtarkov sum and dehned as 


x^X 


(3) 


def 

For notational convenience we denote p(x) = maxpepp(x), to be the maximum probability any 
distribution in V assigns to x. 


4.2 Small alphabet case 

Recall that R(A^) « logn. We now give a simple example to show that unordered distribution 
classes may have much smaller redundancy. In particular we show that for a distribution p 
over k symbols, 

R('P(p)) < log/c! < /clogA; Vn. 

Consider the Shtarkov sum 




< 


E E 

x’^eX’^ peP(p) 


E E 

pG'P(p) x^^X" 


= E i = iP(p)i = v 

PGP(p) 


Clearly for k, the above bound is smaller than ii(A^). 


4.3 Large alphabet regime 

From the above result, it is clear that as n ^ oo, the knowledge of the underlying-distribution 
multi-set helps in universal compression. A natural question is to ask if the same applies for the 
large alphabet regime when the number of samples n <C /c. Recall that Acharya et al. [2014b], 
Boucheron et al. [2009] showed that for power-law envelopes, /(/) = c • with infinite support 
size 

R{£f) = e{n^). 

We show that if the permutation of the distribution is not known then the worst-case redundancy 
is Q{n) S> 0(n«), and thus the knowledge of the permutation is essential. In particular, we prove 
that even for the case when the envelope class consists of only one power-law distribution, R scales 
as n. 


Theorem 1. For n < /c^/", 


> R0^{zipf{a,k))) > ^log 


k — n 
n°^Ck,a 
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we have 


Proof. Since ^ 

To lower bound recall that 

s(P(”ipfKi))) = E#(*”) 

> ^ p{x^), 

x'^:ip'^=n 


where is the number of distinct symbols in x”. Note that number of such sequences is k{k — 
l){k — 2)... {k — n + 1). We lower bound p(x”) for every such sequence. Consider the distribution 
q G ’P(zipf(a,fc)) given by q{xi) = Ml < i < n. Clearly p{x'^) > g(x") and as a result we have 


n 

s(n”ipt(p.t))> > - !)(*: - 2)... (t -»+1) n 

k — n 


i=l 


i^Ck. 


> 


n^Ck, 


Taking the logarithm yields the result. ■ 

Thus for small values of n, independent of the underlying distribution per-symbol redundancy 
is log Since for n < k, R{^k) ~ n\og we have for n < 


R{£(ci-<^,k))<RiK)<0{n\og- 

Therefore, together with Theorem 1, we have for n < kf/^ 


^ ^(^rci-“,fc)) < 0{n\og^). 


n 


5 Expected redundancy based on the number of distinct elements 

In order to find the redundancy of the unordered envelop classes, we follow a more systematic 
approach and dehne another structure on the underlying class of distributions. More precisely, we 
consider the class of all distributions in which we have an upper bound on the expected number of 
distinct elements we are going to observe. Lets define 

Vf = {p-:Ep[pl]<d}, 

where is the number of distinct symbols in the sequence x”. Note that for any distribution 
belonging to this class, all permutations of it are also in the class. We later show that envelop 
classes can be described in this way and the expected number of distinct elements characterizes the 
envelop classes; therefore we can bound the redundancy of them applying results in this section. 





5.1 Upper bound 

The following lemma bounds the expected redundancy of a class in terms of d. 

Lemma 2. For any class V^, 

RCPd) <dlog^ + (21oge + l)d + log(n + 1). 

Proof. We give an explicit coding scheme that achieves the above redundancy. For a sequence x” 
with multiplicities of symbols Fi-:d 2 -, ■ ■ ■ ■, dk-, let 


q{x^) 


1 



H 


be the probability our compression scheme assigns to x” and 
by 



is the normalization factor given 


Before proceeding, we show that g is a valid coding scheme by showing that q{x^) < I- 

We divide the set of sequences as follows. 


n 

E =E E E E 

d'=l S€X:\S\=d' iff x”:ti(x")=ti>‘ 


Now we can re-write and bound q{x^) as the following. 

n / \ n 

E E E E «(-”) < E E 

d' — l S£X:\S\=d' fL^:fLi=0 iff i^S x'^:fi{x'^)=fL^ d' — l S£X:\S\=d' 


0 


E 

d' = l 


a) • (ri) 


Nd' 


n 

E- 

^ n 


= 1 . 


d'=i 


E 

iff i^S 


1 


where (o) holds since for a given , the maximum likelihood distribution for all sequences with 
same values of ^i, yL 2 , ■ ■ ■ are same. Also (6) follows from the fact that the second summation 
ranges over values and the third summation ranges over (^/~^) values. Furthermore for any 


eV 


n 
d ’ 


log 


p(x"') 

q{x^) 


k 

< log + n • ^ — log < log 

+ ^ n pi/n + 
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Taking expectation over both sides 


R{£{f)) < E[logA^<^^] 
k 


< log n + E 


log 


+ log 


(a) 

< log n + E 


y^+log 


n — 1 

,v^+-VJ 

k 2n\ 




+ (21oge)¥5!: 


kn 

< log n + d log — 5 - + (2 log e + l)d, 


where (o) follows from the fact that (^) < and ( 6 ) follows from Jensen’s inequality. 


5.2 Lower bound 

To show a lower bound on the expected redundancy of class V^-, we use some helpful results intro¬ 
duced in previous works. First, we introduce Poisson sampling and relate the expected redundancy 
in two cases when we use normal sampling and Poisson sampling. Then we prove the equivalence 
of expected redundancy of the sequences and expected redundancy of types. 

Poisson sampling: In the standard sampling method, where a distribution is sampled n times, 
the multiplicities are dependent, for example they add up to n. Hence, calculating redundancy un¬ 
der this sampling often requires various concentration inequalities, complicating the proofs. A useful 
approach to make them independent and hence simplify the analysis is to sample the distribution 
n' times, where n' is a Poisson random variable with mean n. Often called as Poisson sampling, 
this approach has been used in universal compression to simplify the analysis Acharya et al. [ 2012 , 
2013], Yang and Barron [2013], Acharya et al. [2014b]. 

Under Poisson sampling, if a distribution p is sampled i.i.d. poi(n) times, then the number 
of times symbol x appears is an independent Poisson random variable with mean npx-, namely, 

Pr(/ia; = p) = - - Mitzenmacher and Upfal [2005]. Henceforth, to distinguish between 

two cases of normal sampling and Poisson sampling we specify it with superscripts n for normal 
sampling and poiin) for Poisson sampling. 

Next lemma lower bounds R{V'^) by the redundancy in the presence of Poisson sampling. We 
use this lemma further in our lower-bound arguments. 

Lemma 3. For any class V, 

Proof. By the definition of 


;^(ppoi(n)) 


minmc^Epoi(„) 

q peV ^ ^ ’ 



Ppoi(n) ) 

q(xp') 


(4) 


where subscript poi(re) indicates that the probabilities are calculated under Poisson sampling. Sim¬ 
ilarly, for every n'. 


R{V^ ) = min max E 
q per 



p{x'^') 
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Let Qn' denote the distribution that achieves the above minimum. We upper bound the right hand 
side of Equation (4) by constructing an explicit q. Let 


n 




n'l 


Clearly g is a distribution as it adds up to 1. Furthermore, since Ppoi{n)ix'^') = e we 

get 


i?(ppoi(n)) < max]Epoi(„) 
pGV 


log 


Ppoi{n){x ) 

q{x^') 


= max e "—rE 
P&V ^ n'l 

n'=0 


< V e“"—maxE 
^ n'l p&V 

n'=0 

°° „n' 

= E )• 


log 

log 


e~''^p{x''') 
e-''^qn'{x''') 

p{x''') 

Qn'ix"') 


n'=0 


where the last equality follows from definition of By monotonicity and sub-additivity of R{V''') 
(see Lemma 5 in Acharya et al. [2012]), it follows that 

RiV"') < 


< 


< 


n 


n 


RiV") 






Substituting the above bound we get 


R{VVoi{n)^^ < ^ 


, n" 


n'=0 


n'\ 


n 


n 


+1 R{V^- 


= 2R{V''), 


where the last equality follows from the fact that expectation of n' is n. ■ 

Type redundancy: In the following lemma we show that the redundancy of the sequence is 
same as the redundancy of the type vector. Therefore we can focus on compressing the type of the 
sequence and calculate the expected redundancy of that. 

Lemma 4. Lets define riV") = {t(p"') : p G V}, then we have 

R{t{V'^)) = RiV"). 
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Proof. 


R(V'^) = min max E 
q p&V 


log 


p(x"') 

q{x'^) 


= mm 


max > pix^) log , 
q PGP ^ ^ q{x^ 


p{x'^' 


= min max > > 

q peP ^ ^ ^ 

x'^£X'^:T{x'^)=r 


min m^ j p{. 


ir^:r(x^)=T 

p{t) 



q{x'^) 

'^x”:t{x”)=t P{x^) 
Sa;":r(x")=r Q{p^) 


= min max > pfr) log 
q pGP ^ 


<l{'r) 


=Rir{Vn) 


where (a) is by convexity of KL-divergence and the fact that all sequences of a specihc type have 
the same probability. ■ 


Now we reach to the main part of this section, i.e. lower bounding the expected redundancy of 
class V^- Based on the previous lemmas, we have 

R{V2) > = lR{r{vr^"^)) 

and therefore it is enough to show a lower bound on R{t{V2°^^^^)). We decompose R{t{V2°''^^^)) 
as 

= min max p(t^) log ^ 


q{T^) 

■■ min max p{p^) log ~~r~ 

log ^ 


q{T^) 


p{t^) 


Hence it suffices to show a lower bound on p(t^) log and an upper bound on '^.^k p{t^) log ^^^k^ ■ 

For the first term, we upper bound q(T^) based on the number of distinct elements in sequence 
^poi(n)_ Lemmas 5, 6, 7 prove this upper bound. Afterwards we consider the second term and it 
turns out that this term is nothing but the entropy of the type vectors under Poisson sampling. 

The following two concentration lemmas from Gnedin et al. [2007], Ben-Hamou et al. [2014] 
help us to relate the expected number of distinct elements for normal and Poisson sampling. We 
continue by a lemma making connection between those two quantities. Denote the number of 
distinct elements in as and = E[<p[|.“^"'^]. Similarly, (p'f is the number of 

distinct elements in x” and d = E[(p!j:]. 
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Lemma 5. (Ben-Hamou et al. [2014]) Let v = he the expected number of elements which 

appeared once in then 

- V^] < e-". 

Lemma 6. (Lemma 1 in Gnedin et al. [2007]) Let Ihe expected number of elements 

which appeared twice in then 


l^poi(n) _ ^1 ^ 

n 


poi(n)-\ 


Using Lemmas 5 and 6 we lower and upper bound the number of non-zero elements in 

Lemma 7. The number of non-zero elements in is more than (1 — e)d with probability 

Also, the number of non-zero elements in < (1 -|- e)d with probability 


d(e-2/n)^ 

> 1 — e 2 

d(e-2/rif 

> I — e 2 


Proof. The number of non-zero elements in r is equal to the number of distinct elements in 
By Lemma 5 

- e)] < 

(a) ^poi(n)^2 

< e 2 , 


where (a) is because > v. Lemma 6 implies — ^) < d < _|_ If Therefore, 




d{l - e)] < PrfypP™^'^) < (^1 + (1 - e)] 


< e 




Proof of the other part is similar and omitted. ■ 

Next, we lower bound the number of bits we need to express based on the number of nonzero 
elements in it. 

Lemma 8. If number of non-zero elements in is more than d', then 

«T) s A;- 

[d'J 

Proof. Consider all the type vectors with the same number of non-zero elements as r^. It is not 
hard to see that q should assign same probability to all types with the same profile vector. Number 
of such type vectors for a given number of non-zero elements d' is at least . ■ 
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Note that the number of non-zero elements in is same as Based on Lemmas 7 and 8 

we have 


^ > E log 7;^ 


> p(r'^)log 


g(r* 


A: 

d(l - e) 




> 1 - e-^ log 


k 

d{l - e) 


= log ( ^ )(1 + Od(l)). 


(5) 


where the last line is by choosing e = d~3. Now we focus on bounding the entropy of the type. 
Recall that if distribution p is sampled i.i.d. poi{n) times, then the number of times symbol i 
appears, pi, is an independent Poisson random variable with mean A* = npi. First we state a useful 
lemma in calculation of the entropy. 

Lemma 9. If X ^ poi{\) for A < 1, then 

77(X) < A[l-logA]+e-^^. 


Proof. 


H[X) = - '^pilogpi 


2=0 

00 


^ X A® e ^A® 

7r‘“s- 


2=0 

00 

i=0 

00 


-A 


A® 


i=0 


,-A 


tl 


A® 


il 

.-A 


loge -b f log A — log(i!) 


A® 


.A® 


=1 [A — A log A] -b e ^ 
< [A-AlogA] + e"^ 


logAj^ze + Elog('’) 

i=0 

A® log(d) 


i=0 

00 

E 

, 2=2 

00 


EA 

. 1=0 

= A[1 - log A] -b e"^ 


where (a) is because the first two terms in the last summation is zero and for the rest of the terms, 
log(z!) < i! Also (b) follows from geometric sum for A < 1. ■ 
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We can write 


k 

i=\ 

k 

= '^H{poi{Xi)) 

i=\ 

= ^ H{poi{\i)) + ^ H{poi{Xi)) 

Xi<0.7 Ai>0.7 

= ^ - Ailog Aj + ^ H{poi{Xi)) 

Ai<0.7 ^ Ai>0.7 

< ^ (3Aj - AilogAi) + ^ ilog r27re(Ai + ^)'j (6) 

Ai<0.7 Ai>0.7 ^ ^ 


where (a) is due to Lemma 9 and (b) is by using Equation (1) in Adell et al. [2010] and the fact 
that < 2x for x < 0.7. 

In the rest of this section, we calculate an upper bound for the second term in (6). Note that 
the hrst term in the same equation, i.e. is heavily dependent on the shape of 

distributions in the class. In other words, upper bounding this term generally, will lead us to a weak 
lower bound, while plugging in the exact values leads to a matching lower bound for the intended 
envelope class, i.e. Zipf distributions. 

Let n~ be the sum of all A < 0.7 and re"*" be the sum of all A > 0.7. Similarly, we dehne k~ and 
A;+ as the number of A < 0.7 and A > 0.7 respectively. Therefore we have 


/c+ ( 1 - 


=-0.7 


E L 


i:Ai>0.7 


=-0.7 


< ^ 1 - exp{-Xi) 

i:Ai>0.7 

< ^ 1 - exp{-Xi) 


(a) 

< d{l + e) 


where (a) follows from Lemma 5 and the fact — exp{—Xi). Hence we have k~^ < 

j/-| I 2 \ r1('\ _L ^ ^ 

-- < 2d{l + -) and consequently k~ > k — - -f^. For the type entropy we know 

"■ l--rr-T 


rUTT 


h + A) ) < log [ 27re( 


1 


Ai>0.7 




12 


(a) 1 




,n' 1 , 

12 ' 


(^) 2 f 

< d{l H —) log 7re( 

n \ 




Ti 1 

= ( 6^ dlogvre ) (1 + 0 ^( 1 )) 


( 7 ) 


where (a) is by concavity of logarithm and {b) is by monotonicity. 
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Lemma 10. 


> flog - dlog - dlogvre'j (1 + Orf(l)) - ^ (3Ai - A* log A*). 

^ ^ ^ ' Ai<0.7 

Proof. (5), (6), and (7) leads to the theorem. ■ 


6 Expected redundancy of unordered power-law envelope 


To use Lemmas 2 and 10 we need to bound the number of distinct elements that appear from any 
distribution in the envelope class in addition to calculating the last summation in Lemma 10. 
For a distribution p £ the number of distinct elements is 

k 

E[ipl] = Y,m,.>o] 

i=l 

k 

i=l 

k 

i=l 

k 

< 1 - (1 - f(i)r 

i=l 

< E 1+ E i-(i-/w)” 

< X] Z] 

r.f{i)<l/n 

Thus we need to bound the number of elements with envelope > 1/n and the sum of envelopes for 
elements that are less than 1/n. For the first term is < (n/c)^/" and the second term is 


< 


E 


cni"" < 


i=(n/c)^^° 


a — 1 


n{njc)' 


Combining these, we get 


d< 


1 


< n^/“. 

a — 1 


+ ^ 


a — 1 

For 'P(zipf(Q!,fc)), we calculate X^Ai<o. 7 '^* Z]Ai<o .7 “-^dog A, for a > 1. In the below calcula¬ 


te 






tions means that the quantities are equal up-to a multiplicative factor of 1 + 0 ^( 1 ) • 

k . — f~v 


n = Aj 

Ai<0.7 


E 


n 




Ck, 


M h \ l/o „ 

lOn \ Ck,, 


7C 


k,a 


-di 


n 


lOn 


— (a—1 )/q 


lOn ^ -(“-!)/“ 




n 


{a-l)Ck,a \7Ck, 


lOre 


lO(a-l) \7Ck,, 

Now we calculate the other summation. For 'P(zipf(a,fc)), 


-AjlogAj 

Ai<0.7 


^ E 


i " , fCk,ai‘' 

n —— log(- 


Ck. 


a 


n 


— -t / Ck OL X 'kiOt 

=n log(- ) + 7=r— 

^ Ck^OL 


E 


i"" logi 


^ _1 tCk,a^ , na 

<n log(-) + 7 ^ // 

n (Ffc.g 10" ) -1 

y ‘^k,a J 


i/a i “logidi 


, _ , , Ck,a X Tia 


“((a — 1) logx + 1) 


(a — 1)2 


Ifcx. 


+ 


na 1 / lOn 


- k 


Ck,a ^ V '^^k, 


-1 


log 


lOn 

7Ck,a 


<n log(—H- 

n Ck,aa-V7Ck, 


1 lOn lOn net / lOn 


X -1 


+ 


< 


+ 


n 


lOn 


Cfc.a \7Cfc, 
7 / 


-1 


log 


lOn 


10(a - 

1) V 

7Ck,aJ 

7a 

f 

lOn 

10(a - 

1)2 1 

7Ck,a^^ 

11.2a - 

-4.2 

f lOn ' 

10(a - 

1)2 

\7Ck,a. 


lOn 
TCk,a 

, /Cka 

log - 

n 


7 lOn 1 lOn 


lOn 


10 °7Cfc,, 


+ T7T log ■ 


lOn 


10 "7Ck,a 

Substituting the above bounds in Lemmas 2 and 10 results in the following theorem. 
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Theorem 11. For k > n, ^ ^ j, c' = ^J, and cs = lo(a-i)^ \7C^ 

c[ ■ log vre 


_ 32.2a-25.2 / 10 


Ri£{ci-<^,k)) > R{^Upf[a,k))) > (log - c'l(l - log^) (1 + 0,(1)) - C 2 


1 7 , lOn 

— C 2 • n«-log ■ 


10 ^7Cfc,„’ 


and 


fc)) ^ log ( 1 ) + ci(2-h 2 log e) • n« log — + log(n + 1) 

'' ’ ' \cina J a c\ 

Wo can write both of the bounds above in order notation as 


^i£(ci-'=‘,k)) = Qin<- log A:). 
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